Noncollinear Magnetic Order Stabilized by Entangled Spin-Orbital Fluctuations 
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Quantum phase transitions in the two-dimensional Kugel-Khomski model on a square lattice are 
studied using the plaquette mean field theory and the entanglement renormalization ansatz. When 
3z 2 — r 2 orbitals are favored by the crystal field and Hund's exchange is finite, both methods give 
a noncollinear exotic magnetic order which consists of four sublattices with mutually orthogonal 
nearest neighbor and antiferromagnetic second neighbor spins. We derive effective frustrated spin 
model with second and third neighbor spin interactions which stabilize this phase and follow from 
spin-orbital quantum fluctuations involving spin singlets entangled with orbital excitations. 
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Introduction. — Almost 40 years ago Kugel and Khom- 
skii realized that spins and orbitals should be treated 
on equal footing in Mott insulators with active orbital 
degrees of freedom Their model explains qualita- 
tively the magnetic and orbital order in KCUF3 which 
is a well known example for spinon excitations in a one- 
dimensional (ID) Heisenberg antiferromagnet [2j. This 
archetypal compound is usually given as an example of 
the spin-orbital physics [3j, which covers a broad class of 
transition metal compounds, including perovskite man- 
ganites [3, titanates vanadates 0, ruthenates Q, ID 
cuprates (H. layered ruthenates and pnictide super- 
conductors [10j]. In all these compounds strong intraor- 
bital Coulomb repulsion U dominates over electron hop- 
ping t (t <C U) and charge fluctuations are suppressed. 
On the one hand, spin degrees of freedom may sepa- 
rate from the orbitals when the coupling to the lattice 
is strong, as in LaMn03 [4| and recently shown to hap- 
pen also in KCUF3 |ll|. On the other hand, the spin- 
orbital quantum fluctuations are strongly enhanced for 
low S = i spins, as in the three-dimensional (3D) Kugel- 
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Geometrical frustration [14| 



was also suggested as a stabilizing mechanism for a spin- 
orbital liquid phase 1151 ■ with examples on a triangular 
lattice in e g (LiNiOo (lq) and t 2g (LiNi0 2 0) orbital 
systems. Frustrated spin-orbital interactions to further 
neig hbors may also destabilize long-range magnetic order 
[18| . An opposite case when orbital excitations determine 
the spin order was not reported until now. 

The phase diagram of the 3D KK model remains con- 
troversial in the regime of strongly frustrated interactions 
— it has been suggested that either spin-orbital fluctua- 
tions destabilize long-range spin order Il2i|. or an orbital 
gap opens and stabilizes spin order |13| . This difficulty 
is typical for systems with spin-orbital entanglement [19j 
which may occur both in the ground state [20| and in 
excited states [2lJ. The best known examples are the ID 
[2^ or two-dimensional (2D) SU(4) models, where 
spin and orbital operators appear in a symmetric way. 
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Figure 1. (color online). Two variational ansatze used in the 
present paper: (a) PMF, and (b) ERA. Black dots are lattice 
sites, Vs are variational wave functions on 2 x 2 plaquettes, 
and Ws are variational 2x2 unitary disentanglers. 



Instead, the symmetry in the orbital sector is much lower 
and orbital excitations measured in KCUF3 [24| are ex- 
pected to be inherently coupled to spin fluctuations [HI . 

In this Letter we present a surprising noncollinear spin 
order in the 2D KK model which goes beyond mean field 
(MF) studies [2^, and explain its origin. So far, non- 
collinear spin order was obtained for frustrated exchange 
in Kondo-lattice models on square lattices, without [27] 
and with J28| orbital degeneracy, or at finite spin-orbit 
coupling [29(| . In MnV2 04 spinel it is accompanied by a 
structural distortion and the orbital order [30(. Here we 
find yet a different situation — when frustrated nearest 
neighbor (NN) exchange terms almost compensate each 
other and orbitals are in ferro-orbital (FO) state, the spin 
order follows from further neighbor spin interactions trig- 
gered by entangled spin-orbital excitations. 

Variational approach. — We begin with presenting two 
general variational methods for spin-orbital systems: 

(i) the plaquette MF (PMF) ansatz, see Fig. HJa), and 

(ii) the entanglement renormalization ansatz (ERA) [jjH , 
see Fig. [ljb). In the PMF, adapted here from a simi- 
lar method for the bilayer KK model [32j, one employs 
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a variational ansatz in a form of a product of plaquc- 
tte 2x2 wave functions P's [33||. Energy is minimized 
with respect to P's to obtain the best approximation to 
the ground state. ERA is a refined version of the PMF, 
where the product of P's is subject to an additional uni- 
tary transformation, being a product of 2 x 2 "disentan- 
glers" Li. They introduce entanglement between different 
plaquettes and make ERA more accurate. 

Kugel-Khomskii model and methods. — The perturba- 
tion theory for a Mott insulator with active e g orbitals in 
the regime of t <C U leads to the spin-orbital model [3J|, 
with the Heisenberg SU(2) spin interactions coupled to 
the orbital operators for the holes in the d 9 ionic states, 
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Each bond (ij) connects NN sites {i,j} along one of the 
orthogonal axes 7 = a,b in the ab plane. The model de- 
scribes the spin-orbital superexchange in K2CUF4 [35J, 
with the superexchange constant J = 4i 2 /U. The coeffi- 
cients n = 1/(1-37?), r 2 = r 3 = 1/(1 — 77), r i = l/(l + r ?) 
refer to the dfd 9 ^ d$dj° charge excitations to the up- 
per Hubbard band [34j and depend on Hund's exchange 
parameter 
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The spin projection operators IT^ and ily 
(II|-) or triplet (Ily) configuration for spins S 
the bond (ij), respectively, 
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Here act in the subspace of e g orbitals occupied by 
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matrices {erf , erf} in the following way [3 
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The term H in Eq. (1) is the crystal field splitting of two 
e„ orbitals induced by the lattice geometry or pressure, 
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When |J5 a | ^> J it dictates the FO order with either z 
or x orbitals as long as we stay in the AF regime. This 
ground state can be further improved using perturbation 
theory in a dimensionless parameter |e 2 | _1 = J/\E Z \. 



In the PMF one finds self-consistently MFs: 
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Figure 2. (color online). Phase diagram of the 2D KK model 
in the PMF (solid lines) and ERA (dashed lines) variational 
approximation. Insets show representative spin and orbital 
configurations on a 2 x 2 plaquette — x-like (t c — i) and z-like 
(t a,c — — i) orbitals [3^ | are accompanied either by AF spin 
order (arrows) or by spin singlets in the PVB phase (ovals). 
The FM phase has a two-sublattice AO order (with t a(b) = § 
at E z = 0) or FO2 order (FMz). In between the AF and FM 
(FMz) phase on finds an exotic ortho-AF phase — it has a 
noncollinear spin order, see text. 



and i = 1, ■•• ,4 labels sites of a single plaquette, see 
Fig. [TJa). We assume that either all plaquettes are 
the same, or that the neighboring plaquettes are rotated 
by 7r/2 with respect to each other in the ab plane. In 
the latter case the order parameters are interchanged 
(a O b) between neighboring plaquettes and transform 
as: {tf b) } -> {^ {a) } and {v a(fc) } {v? Ma) }. In the 
ERA treatment we either assume that all "P's and Li's 
are the same, like in Fig. [T^b), or divide the plaquette 
lattices of P's and W's into four sublattices with four in- 
dependent P's and Ws. One finds that the energy found 
in the ERA, when optimized with respect to both hi and 
P, is typically 5. ..15% lower than the one in the PMF. 

Phase diagram. — The phase diagram in [e z , rj) plane 
contains six phases, see Fig. [5] The same phases appear 
in both the PMF and ERA — this suggests that the 
phase diagram is complete. At large 77 one finds two 
FM phases: either with alternating orbital (AO) order as 
observed in K 2 CuF 4 [H OH or with FOz order (FMz). 
At E z < —1.8 J a second order transition occurs from 
the FM to the FMz phase (all other transitions involve 
both spins and orbitals are first order) at e z /r-y — —0.934 



(0.837) in the PMF (ERA). In these phases Ilf^ = 1 and 
the Hamiltonian ([1]) reduces to the e g orbital model (39| 
or to the generalized compass model [40], in transverse 
field E z . At E z = one finds AO order with {er x ) ^ 0, 
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Figure 3. (color online). Schematic view of two nonequivalent 
spin configurations (a) and (b) of the classical ortho-AF phase 
|AFj_) which cannot be transformed one into the other by lat- 
tice translations. Four spin directions (arrows) correspond to 
four spin sublattices — up/down arrows stand for eigenstates 
of (Sf) = ±5, while right/left arrows for (Sf) = ±±. 



while finite E z induces transverse polarization (a z ) =/= 0. 

The phase diagram includes also two AF phases. They 
have uniform FO order with (a z ) > for E z > and 
(<7 Z ) < for E z < 0. The spin interactions in two AF 
phases are nonequivalent and are much weaker for E z < 
than for E z > — this difference increases up to a factor 
of 9 for fully polarized orbitals [4l[ . These two phases are 
separated by the plaquette valence bond (PVB) phase 
with pairs of parallel spin singlets, horizontal or vertical 
and alternating between NN plaquettes. Note that the 
PVB phase is an analogof spin liquid phases found before 
for the 3D KK model and for the bilayer 

Finally, at E z < and rj ~ 0.15 we find a novel ex- 
otic "orthogonal AF" ("ortho-AF") phase with entangle- 
ment (v" ' 7 ^ 0) which emerges in between the AF and 
the FM (FMz) phase. This state is characterized by the 
noncollinear magnetic order, see Fig. [3j with NN spins 
being orthogonal to each other and next-nearest neigh- 
bor (NNN) spins being AF. This phase is robust and has 
a somewhat extended range of stability in the ERA. In 
contrast to the frustrated spin J1-J2 interactions on a 
square lattice [42|, one finds here that J\ is negligible 
and spin order follows from further neighbor couplings. 

Effective spin model. — To explain the exotic magnetic 
order in the ortho-AF phase shown in Fig. [3] we derive an 
effective spin model for this phase. We show that NNN 
and third NN (3NN) spin interactions emerge here from 
the frustrated spin-orbital superexchange, V = H — Ho, 
treated as perturbation of the orbital ground state |0) of 
the unperturbed Hamiltonian Ho (0). Note an analogy 
to hidden multiple-spin interactions derived recently for 
frustrated Kondo lattice models [43j. 

For negative e z < the ground state |0) of Ho is the 
FOz state with z orbitals occupied by a hole at each site, 
T i |0) — ~\ |0), and the energy e = ~^l e z| per site. 
A finite gap that occurs for orbital excitations helps to 
remove high spin degeneracy in |0) by effective spin inter- 
actions in the Hamiltonian that can be constructed 



Figure 4. (color online). Artist's views of the effective spin 
interactions obtained in: (a) second order Hg , with NNN 
term (123) and 3NN term (124), and (b) third order H ( a 3) . 
The frames in (a) indicate Heisenberg bonds multiplied along 
a or b axes with ± sign depending on the bond direction; the 
dot in the center stands for an orbital flip in |0). In (b) the 
dashed lines symbolize sums of three spins which enter each 
effective spin, Sjv^i) and Sjv', (jj; the phase factors s 7 (circles) 
and their scalar product are marked with connected frames. 



using the expansion in powers of \e z [ 
J^Neo + H^ +HW 



(6) 



where N is the number of sites. The first order term is 
an average HP = (0|V|0). Similarly, to evaluate HP 
and HP we determine the matrix elements (n|V|0) for 
the excited states \n) with certain number of z-orbitals 
flipped to a;-orbitals. All the averages are taken between 
orbital states and the spin model Eq. ([6]) follows. 
The first order yields the Heisenberg Hamiltonian 
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The NN interaction Ji = (— 3ri + 4r 2 + r^) J/2 5 changes 
sign at r/o — 0.155 implying a direct AF-FM transition. 
However, this turns out to be a premature conclusion 
because the vanishing of hP at r/o makes higher order 
terms in Eq. ^ relevant. Indeed, rjo nicely falls into the 
ortho-AF area of the phase diagram in Fig. [2 where the 
NN interaction J\ is small and frustrated. 

Higher order terms. — Higher order terms arise by flip- 
ping orbitals from the ground state |0). Given that V has 
non-zero overlap only with states having one or two NN 
orbitals flipped from z to x, one finds in second order, 
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with £(77) = (ri+2r 2 + 3r 4 ) 2 /2 10 . Here ((ij)) and ({{ij))) 
stand for NNN and 3NN sites i and j, see Fig. QJa) for 
the origin and sign of these interactions. Apart from this, 
the second order also brings the |£ 2 | _1 correction to the 
Heisenberg interactions of hP ijTj). moving the transition 
point from 770 to tjq + O (e z l \ 



4 




Figure 5. (color online). Artist's view of the ortho-AF state 
l^so) (|10|) . with spin order (arrows) of Fig. [3jand FOz orbital 
order (circles) in the ab plane. The state is dressed with 
spin singlets (ovals) entangled with either one or two orbital 
excitations from \z) to \x) orbitals (clovers) on NN bonds. 



The NNN AF interaction in © alone would give 
two quantum antiferromagnets on interpenetrating sub- 
lattices [HI, but the additional 3NN FM term makes 
these AF states more classical than in the 2D Heisenberg 
model [45]. This "double- AF" configuration is already 
similar to the ortho-AF phase in Fig. [3J However, the 
second order does not explain why the spins in the ortho- 
AF phase prefer to be orthogonal on NN bonds, and we 
have to proceed to the third order. 

The third order in Eq. ([6]) produces many contribu- 
tions to the spin Hamiltonian, but we are interested only 
in qualitatively new terms comparing to the lower or- 
ders. The terms bringing potentially new physics are the 
ones with connected products of three different Heisen- 
berg bonds [4|| . The final result is a four-spin coupling, 
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where 7 = -7 and S A ^ (i ) = J2 a ^j s aSi+a is an effective 
spin around site i in the direction 7, see Fig. Sib). Here 
x(t]) = 9(fx +r 4 )/2 7 , a € {±a,±&}, and s a = —1 for 
a = ±6 and s a = 1 otherwise. In the limit of two in- 
terpenetrating classical antiferromagnets H± gives the 



energy per site, e 



(3) 



2 xMtinM cos ip) 2 , where <p is 



an angle between the NN spins [45] . This classical en- 
ergy is minimized for ip — tt/2 which explains the exotic 
magnetic order in the ortho-AF phase, shown in Fig. [3] 
Spin- orbital entanglement. — The ground state |AFj_) 
of H s ([6]) is nearly classical, except for small quantum 
corrections obtained within the spin- wave expansion [45j . 
Thus one might expect that the spins are not entangled 
with orbitals. However, this argument overlooks that 
the resulting spins in H s are dressed with orbital and 



spin-orbital fluctuations. Indeed, within the perturbative 
treatment we obtain the full spin-orbital ground state, 
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(10) 

where V n = \n){n\V, £ n are excitation energies, and 
|$o) = I AFj_) |0) is the disentangled classical state (Fig. 
[3]). The operator sum in front of |4>o) dresses this state 
with both orbital and spin-orbital fluctuations. When 
the purely orbital fluctuations are neglected and density 
of spin-orbital defects is assumed to be small, one finds 



|^so) - exp 



-— V V 

«rl ^ 



l*0> 



(y)ll7 



where 
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(12) 



is the spin-orbital excitation operator on the bond (ij), 
with A = 3(ri + r 4 )/2 6 and B = y/3(n + 2r 2 + 3r 4 )/2 5 . 
Both terms in Eq. (fT2")l project on a NN spin singlet, but 
the first one flips two NN z-orbitals while the second one 
generates only one flipped orbital. In short, the exponent 
e -.D/l e *l dresses the classical ortho-AF state |AFj_) in 
Fig. [3J with the entangled (spin-singlet/flipped-orbital) 
defects, see Fig. \5\ The density of such entangled defects 
increases when |e z | is decreased towards the PVB phase. 

Topological defects. — The order parameter of the 
ortho-AF phase has non-trivial topology. The ground 
state is degenerate with respect to different orientations 
of its order parameter that consists of two orthogonal 
unit vectors defining orientation of each antiferromag- 
net. The first vector lives on the whole sphere S 2 , but 
the second one is restricted to a circle S 1 because it is 
orthogonal to the first. In addition to spin- wave excita- 
tions, this S 2 x S 1 topology allows for skyrmions (tex- 
tures) [iU and Z2-vortices (hedgehogs) as two types of 
topological defects. The hedgehog is stabilized by the or- 
thogonality of the antiferromagnets. For instance, when 
one of them has fixed uniform orientation of its Neel or- 
der in space, the orthogonal orientation of the other one 
is free to make a hedgehog-like rotation. 

Summary. — We have found surprising noncollinear 
spin order that arises from the NN spin-orbital superex- 
change when ferromagnetic and antiferromagnetic inter- 
actions almost compensate each other in the 2D KK 
model away from orbital degeneracy It is stabilized 
by further neighbor spin exchange generated by entan- 
gled spin-orbital fluctuations which involve spin singlets 
and orbital flips. Similar mechanism works in the 3D 
KK model where it leads to a rich variety of spin-orbital 
phases to be reported elsewhere. 

Finally, we note that magnetic order in spin-orbital 
systems may be changed by applying pressure (47| - 
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indeed a transition from ferromagnetic to antiferromag- 
netic order was observed in K2CUF4 (37l . [38| . Such a tran- 
sition is also found here for a realistic value of r\ ~ 0.15, 
and one could induce it in the antiferromagnetic phase by 
external magnetic field. Whether the antiferromagnetic 
order could be noncollinear as predicted here remains an 
experimental challenge. 
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SUPPLEMENTAL MATERIAL 

This supplement presents the technical details of the 
analysis employed in the paper. We first consider the 
third order terms in the perturbative expansion in sec- 
tion A. They justify the angle ip = ir/2 obtained for the 
nearest neighbor (NN) spins in the regime of the non- 
collinear orthogonal antiferromagnetic (ortho-AF) phase. 
In section B we develop the spin-wave theory for the 
ortho-AF phase and calculate the quantum corrections 
to the order parameter. These calculations show that 
the noncollinear ortho-AF phase is stable with respect to 
the Gaussian fluctuations and the quantum corrections 
are here weaker than for the two-dimensional (2D) AF 
Heisenberg model. 



Third order terms in the perturbative expansion 

The second order in the perturbation theory in |e 2 | _1 
results in two antiferromagnets on interpenetrating sub- 
lattices, but the angle ip between the nearest neighbor 
(NN) spins remains undetermined, see Fig. Ufa). Thus 
we have to consider third order contributions to the ef- 
fective spin Hamiltonian H s of the form: 

z W 

x ]T [J ra (ry)S 4 -S J+7 +A— (rj)} 

7#7' 
7"# — 1 

x Y, S Y [-J xz (v)Si-Si+-y>+K xz (r))} 

7'#7 

x v[-^)Si+7-Si +7+7 "+^(77)](13) 

7"#-7 

where: J xz (tj) = 2~ 5 V3 (n+ 2r 2 + 3r 4 ), K xx {rj) = 
2- 7 v / 3(2r 2 -3ri +3r 4 ), J xx (r]) = 2~ 5 3 (ri + r 4 ) and 
K xx {r\) = 2~ 7 3 (3ri — r 4 ). Here and in all other equa- 
tions in this Section a sum over 7 means the sum over all 
directions in the square lattice, i.e., 7 = ±o,±6. 

The spin chains with less than three scalar products do 
not contribute with any qualitatively new terms. Once 
they are omitted we obtain: 



H. 



(3) 



chain 2 16 £ 2 



(r 1 +r 4 )(r 1 +2r 2 + 3r 4 ) 2 J2 



i,7 iVi 



\2SrySry" [Si-\-ry " Si) (Si " S j -)--y ' ) ( S ' " S 

-j-Sj'Sj" (Si ' Sj-(--y) (Sj-f- J' ■ Sj)(Sj 



i+7'+7" 



+7 ' a i+ 7 + 7 ' 



)}, 

(14) 




Figure 6. Panel (a): two independent AF orders realized 
by effective spin Hamiltonian H„ up to second order. Angle 
ip between the two neighboring spins is undetermined. Panel 
(b): exemplary third order correction to H s fixing the angle 
ip as tp = 7r/2. Red frames stand for Heisenberg bond with 
± sign depending on the bond's direction and magenta dots 
indicate the single-site orbital excitations in the ground state. 



where s 7 is a sign factor depending on bond's direction 7 
originating from the definition of operators r"' 6 [see Fig. 
Mb)}, i.e., 



1 if 7 = ±a 
-1 if 7 = ±b 



(15) 



We transform the second term of Eq. (1141) using the 
vector identity: 

(Sj ■ Sjj_|_ 7 ) (Sj-py' ' Si) = 

(Si+y • Si) (Si • Si +7 ) + iSi +7 (Si + y x Si). (16) 

The antihermitian term with a cross product cancels out 
under the sum in Eq. (|14[) . thus we obtain: 



H 



(3) _ 9 

chain 2 16 £ 2 



(ri+r 4 )(n+2r 2 + 3r 4 ) 2 ^ ^ 

2,7 iVl 



{2S-yS-y" (Si-fry " S^)(Si " S^-|--y' ) (S^-fry' ' S^-fry t _|_^y It J 

-\-S~y'S.y" (Si-^ryf • Si) (Si • Si-f^y) (S^-fry ' S-j-|-*y-f *y'f ) } ■ 

(17) 

Now all the scalar products are ordered along the lines: 
(i + 7) -> (i) -> (i + 7') ->• (i + 7' + 7") and (i + 7') -s- 
(i) — > (i + 7) — > (i + 7 + 7"). Next we use another spin 
identity, namely 

(Si • S 2 ) (S 2 • S 3 ) (S 3 • s 4 ) = 

4 (Si • S 4 ) + \ (Si ■ S 4 ) (S 2 ■ S 3 ) - \ (Si • S 3 ) (S 2 ■ S 4 ) 
16 4 4 



4 Si (S 3 x S 4 ) + |s x (S 4 x S 2 ). 



(18) 



Again, the antihermitian cross-product terms cancel out 

(3) 

under the sums in -ffc nain - 

To analyze the relevance of other terms in Eq. (fT8|) 
we have to take into account that, to second order in the 
perturbation theory, there is nearly classical AF order on 



7 



the two sublattices. We observe that: 

(i) the first term is an AF interaction between the sub- 
lattices which is not compatible with the antiferromag- 
netism on the sublattices that is 0(E Z ) stronger, 

(ii) depending on its sign the second term may favour 
orthogonality of the two AF orders which is compatible 
with the order on sublattices, and 

(iii) the third term brings no new information about the 
order. 

Taking into account all three above arguments we ar- 
gue that the relevant type-(ii) third order perturbative 
contributions of the form given by Eq. (TT51) may favour 
orthogonality of the two AF orders. Now we have to ex- 
tract all such contributions from Eq. (fT8)) and check if 
their overall sign is indeed positive. 
After transforming Eq. (jTTJ) we obtain 



H 



(3) _ 



9 



2 i8 £ ; 



<i7 iVi 
7" 



[2SjSj" (Si + y ■ Si_f_-y'_|_-y" ) (S^ ' Sj + y) 

S 7 'S 7 " (S^+y • S.;_|_ 7 ^y )(Sj • Sj+ 7 )] , (19) 



or in a more compact form 



H 



(3) 



27 



(ri+r 4 )(r 1 + 2r 2 +4r 4 ) 2 ^(S i .S i 

i,7 



1 • (20) 

(3) 

For two interpenetrating classical antiferromagnets H\_ 
gives the energy per site, 

e¥*^xmtn)(l cos^) • (21) 

This classical energy is minimized for ip = ir/2, i.e., when 
the NN spins on the bonds are orhogonal. This completes 
the argument that the orders in the two antiferromagnets 
prefer to be orthogonal. 



Spin wave expansion in the noncollinear ortho-AF 
phase 

We start from the general form of the effective spin 
Hamiltonian: 

H s =A^2Si ■ Sj + 2B ^ Si ■ Sj - B ^ S t • Sj 

(ij) ((ij)) ««»> 

+ C ^ ^ (Si • Sj) ^ ^ SySy/ (Sj_py' • S^-y+y) , 
<*i>ll7 



7 #7 
l" — 7 



(22) 



with coeffcients A, B and C being the functions of r\ and 



e«, i.e. 



A = — (-3n + 4r 2 + r 4 ) 
1 3 



+ -gil { 3 W - r ') - 2 (n + 2r 2 + 3r 4 ) 2 } , (23) 

S = --^ T (ri+2r 2 +3r 4 ) 2 , (24) 
27 

C = -j^-j (n + r 4 ) (n + 2r 2 + 3r 4 ) 2 . 

Z 6 ~ 



(25) 



To describe the ortho-AF order we divide the lattice into 
four sublattices as follows 



= &p+2i,q+2ji P,q S {1,2}, 



(26) 



where p and q form the sublattice label. In what follows 
all the sums over p and q run over the set {1,2}. Now, in 
each sublattice we do the linearized Holstein-Primakoff 
transformation around the ortho-AF order, i.e., 

= K«S t+0 &)> - s 'f § («;: • «8) • 



f' 22 - J- f „ 22 t _ „22 



(27) 



and 



jx,12 _ / 12t 12 _ 1 A 9 *,21 _ A _ 21f 21 A 
~ ^"ij' "ij 2 J ' — \^2 U i,3 U i,j)> 



,12 _ l/„12t , 12^ oZ/,21 _ 1/ 21 + , 21 

S- 2 - 12 - J-/V 2t - n 12 ^ S z > 21 - if a 21 - n 21t \ 
D i,j ~ 2i\ a i,j "■■:) ' ~ 2-\ ' ,' J 



Next step is the Fourier transform (FT): 



i(ik b +jk a ) pq 
"k > 



(28) 



(29) 



followed by the phase transformation, in order to get rid 
of imaginary parts in H s of Eq. (|2"2"|) after FT, 



-, n t , „"t„-i% 



Oi, — t la 



al 1 -> -iaJV^e"**, a?* -> a^V*. (30) 



fc > 

22f 22f 



Finally, the interactions in the i? s Hamiltonian take the 
following form: 

5> . Sj = \y: c ° s y - <%) (4 if - « n . 

(ij) k 

1^ fcfe/ 12+ 12 \ / 22+ 22 

+ 2^ yr- k ~ ° k J r k a - k 

k 

^a/ 12+ 12\ ( lit 11 

_ 2^ cos Yr- k k ) r k - k 

a / 21+ 2l\ / 22+ 22 

j-^a-k-flk J (a k '-o_i 



1 k„ 
- > cos ■ „ 
2^ 2 

k 



(31) 



for the NN interactions, 



E S, • S, = 2 £ { 7 L V 2 t + ^t^t) + K.c.j 

k 

+ 2EE< 9t <- (32) 



<(ij» 



k p,«j 

for the NNN interactions, 

E S .- S ) = 2 EE<'V , (7 k -l) (33) 

(({i,3))) k p,g 

for the 3NN interactions, and 
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(34) 



for the third order interactions between the two AF sub- 
lattices. The coefficients {7k,7kl are defined as: 



7k = ^ (cosfc b + cosk a ) , 

/ 1 I k a + kj, k a — Kb 

7k = - cos h COS 

k 2 V 2 2 



(35) 
(36) 



The last step is the Bogoliubov transformation of the 
block-diagonal Hamiltonian H s in the momentum space. 
The most general form of this transformation is: 



b k J \ a_ k 



(37) 



Here = {t^ 1 ^} are the new boson operators and Bk 
is an 8 x 8 transformation matrix to be determined from 
the Bogoliubov-de Gennes eigenequation, 

[H s x qi ] = K q K 9t , (38) 

equivalent to an 8 x 8 matrix eigenproblem with hyper- 
bolic normalization conditions typical for bosons. The 



0.41 - 




Figure 7. The sublattice magnetization (S^j 11 ) as function of 
Ez in the ortho-AF phase for: rjo constant such that A (r/o) = 
for e z — > —oo (lower-red curve), and 770 (e z ) such that A (770) = 
for any e z (upper- blue curve). The inset shows the function 
r;o(Ez) with its asymptote for e z — > —00 marked with the 
black dashed line. 



resulting eigen- vectors form the rows Sk and the excita- 
tion energies E™ can be expressed analytically as, 

(E™) 2 = 2 4 (1 - t£« ) {B - 6C + (B - 24C) 7 f } 
x { 3 S + A7r- J B(7k-2(-l)^7k)} ) 



with 



V P9 _ 



- k =§U-l) p cos^ + (-!)« cos I 



(39) 



(40) 



As might have been expected, in addition to the three 
gapless Goldstone modes for (p, q) — (2, 2) , (1, 2) , (2, 1), 
there is one gapped branch E^ 1 , with E^ = 
24-^/ (AB — A) C, related to the rigidity of the angle ip 
between the two antiferromagnets. 

Another important quantity is the magnetization on 
a sublattice which quantifies quantum fluctuations. For 
instance, the ground state expectation value of S*'^ 1 can 
be expressed by the elements of 1 , 



S: 



p=5 



(41) 



The integrand in the above formula can be obtained an- 
alytically whereas the integration is non-algebraic. In 
Figs. [7] and [S] we show the behavior of (^S*^ 1 ^ along 
different cuts of the ortho-AF phase. 
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Figure 8. The sublattice magnetization (S^'j 1 ) as function 
of r\ in the ortho-AF phase for e z = — 10 (solid line). Dashed 
line marks the value of r/o where A = 0. The vertical solid 
(red) lines are boundaries of the spin-wave expansion, where 
the integral (|41[) becomes divergent. 



Both cuts in Fig. [7] take the same value of yS'j 
0.385 in the limit of e z — > — oo when the coupling between 



the two antiferromagnets is negligible. This magnetiza- 
tion is markedly higher than that for the 2D AF Heisen- 
berg model, where (S z ) ~ 0.303. This confirms that the 
NNN FM interactions in H s make the AF order more ro- 
bust against quantum fluctuations. What is more, along 
the line rjo{e z ) (blue curve) the third order term oc C en- 
hances the order parameter up to (^S*^ 1 ^ ~ 0.405 near 

s z = —4. Deviations from the path t)q (s z ) that introduce 
nonzero coefficient A of the NN AF coupling can either 
impair or enhance the ortho-AF order. For instance, the 
red curve lies below the blue one in Fig. [JJ In contrast, 
the cut along the line e z = — 10 in Fig. [8] shows that a 
small increase of 77 above r]o can increase the magnetiza- 
tion (at this value of e z ). This reflects the proximity to 
the ferromagnetic phase where the quantum fluctuations 

reducing the order parameter (^Sif 1 ^ are suppressed. 

The sudden collapse of the blue and red curves in Fig. 
[7J terminates the ortho-AF phase in the spin wave ap- 
proach. However, this is not a definitive conclusion be- 
cause the perturbative Hamiltonian H s underlying the 
spin wave expansion is not self-consistent for e z > —4, 
where the third order term oc C in H s dominates over 
the second order term ot B. In the non-perturbative pla- 
quette MF the ortho-AF phase extends far beyond the 
perturbative regime, and it is even more extended in the 
more accurate ERA approach. 



